feat(metrics): emit conserver.ingress_list.length observable gauge - #173

Summary

Add an OpenTelemetry observable gauge conserver.ingress_list.length that samples Redis LLEN for every configured ingress list and its derived DLQ. Each observation carries:

Splitting kind into its own attribute (rather than baking DLQ: into the value) lets queries filter live vs dead-letter cleanly: conserver.ingress_list.length{kind="dlq"} > 0 instead of regex-matching ^DLQ: in a label value.

The callback re-reads chain config on every export tick, so new ingress lists begin emitting series automatically — no worker restart required.

Pairs with the conserver.dlq.count counter from #171: counter records the event of moving a vCon to DLQ; gauge records the state of how many are sitting there right now.

Implementation

• New common/lib/queue_metrics.py owns the callback construction (one observation per (ingress_list, kind) combination).
• New register_observable_gauge() helper in common/lib/metrics.py keeps OTel SDK plumbing in one place and matches the existing increment_counter / record_histogram lazy-init pattern.
• worker_loop() calls register_ingress_list_length_gauge() once per worker process, immediately after the Redis client + VconQueue are initialized.
• With N replicas you get N identical series per dimension combination; query-time aggregation (max by (ingress_list, kind)) collapses them.
• Individual LLEN failures (transient Redis errors) skip that series only — the export tick still publishes the series that succeeded.

Test plan

• New common/tests/test_queue_metrics.py covers the callback in isolation (no OTel SDK):
  • empty config → empty observations
  • dedup + sort of ingress lists across multiple chains
  • config-read failure → no series, no exception (degraded mode)
  • one observation per (ingress_list, kind) with correct values
  • ingress_list attribute value is the bare configured name for BOTH the live and DLQ observations
  • individual LLEN failure skips that series, others still emit
• Existing common/tests/test_dlq_expiry.py, test_queue_dlq_counter.py, test_parallel_processing.py still pass
• Manual verification on a running deployment: roll the new image, confirm conserver.ingress_list.length appears with the expected dimensions

🤖 Generated with Claude Code